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Abstract 

A relatively new mobile text message service, ChaCha, describes itself as “a smart search engine powered by 
human intelligence. ” The sendee claims to provide high-quality, accurate information, yet there is no research 
published to date substantiating this claim. The purpose of this study was to assess the extent to which health and 
wellness-related information disseminated by ChaCha is accurate and complete. The study consisted of texting 40 
pre-determined health-related questions to ChaCha, compiling answers, and analyzing answers for accuracy and 
completeness. Using a Likert-type rubric, accuracy was determined by comparing ChaCha responses to three of the 
most well-known and popular college health textbooks. Results indicated only 12.5% of ChaCha responses were 
rated accurate and complete, while 22.5 % of ChaCha 's responses rated completely inaccurate/incomplete. In 
summary, while ChaCha provides a fast, efficient method to find answers, the accuracy and completeness of the 
answers is questionable and not consistent with evidence-based research. Translation to Health Education Practice: 
Health educators can use the results of this study to alert the public that ChaCha is not a reliable source for health 
and wellness-related information. 
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Introduction 

From simple questions to complex, answers are now 
just a few key-strokes away. The internet serves as a 
reference for just about everything, from checking 
scores of ballgames to assisting in major medical 
decisions. Search engines, such as Google and 
Yahoo, provide a vast number of sites to sift through 
to find information. The question that must be asked 
is whether such information is accurate. 

Along with the rise of the internet, cell phone usage 
is also skyrocketing. A study conducted by 
EDUCASE Center for Applied Research found that 
nearly 85 percent of all college students carry a cell 
phone. 1 Originally intended as wireless phone 
service, cell phones have evolved greatly to have the 
capability to download music and ringtones, surf the 
net, and send text and photo messages. Although 
EDUCASE reports only 19% of Americans use 
smartphones that allow for downloading learning 
material, 1 that number is likely to increase 
substantially. With the combination of the internet 
and cell phone usage, search engines including 
Google have initiated computer-automated text 
message answer services. 2,3 

A relatively new mobile text message service, 
ChaCha, describes itself as “a smart search engine 
powered by human intelligence”. 3 ChaCha is a 
relatively free service (meaning you only pay if your 
cell phone service charges a text fee) powered by 
guides who quickly navigate and filter through 
websites to find an answer. The value of ChaCha is 
that consumers who do not have access to the internet 
can quickly use their cell phone to find an answer to 
virtually anything by texting their question to 
ChaCha (242242). The service claims to provide 
only high-quality, accurate information, yet there is 
no published research to date substantiating this 
claim. 

Literature Review 

Accuracy of health education information 
disseminated via the World Wide Web 

The Internet has changed the way Americans obtain 
information and make personal decisions. The Pew 
Internet Project estimates that 74% of American 
adults use the Internet. According to the December 
2007 survey, 75% search online for health 
information. 4 This estimate has grown from January 
2001, where 55% of American adults with Internet 


access have used the Web to get health or medical 
information. 4 

Concern remains evident through various studies 5,6 
regarding the quality of information, and the 
credibility of Web sites which disseminate health 
information. Additionally, as the Internet evolves and 
grows, so does the number of people searching for 
health information. Health practitioners receive 
numerous questions from patients regarding 
information they have encountered online. 

Integration of online sources with patient education is 
a new area of development for some practitioners, but 
challenges remain in guiding patients to reliable and 
accurate web-sites. 7 

The evaluation of health information on the Internet 
has been recognized by various groups and guidelines 
have been developed, 5,8 however, a specific set of 
standards is yet to exist. 9,111 The National Institutes 
on Health has created a specific list of guidelines for 
ensuring the quality of information disseminated to 
the public online. 1 1 Healthy People 2010 has 
incorporated a new goal of communication 
objectives, a measure of increasing adequate quality 
information on the Internet. 12 

In December 2007, a group of eight pediatric 
dermatologists examined the accuracy and 
completeness of information on a specific topic by 
searching Google. Only four sites of 50 evaluated 
were considered accurate and complete enough for 
the dermatologists to recommend them to their 
patients. The results of that study are consistent with 
others that have viewed medical information on the 
Web. 13 

A study published in the May 22, 2002, Journal of 
the American Medical Association reviewed the 
quality of consumer health information on the World 
Wide Web. Published and non-published studies 
which evaluated accuracy of online health 
information were both reviewed. The discrepancy 
between study methods and quality criteria concludes 
that there needs to be defined quality criteria to 
accurately review the quality of health- related 
websites. 14 The quality of health information 
disseminated by the Internet continues to be a 
concern for public health. 7,13 ' 14 

It is reported that 89% of Internet users make use of 
search engines to find information. 4 The massive 
array of search engines can be perplexing to the 
common web user. Yahoo, developed in 1994, 
remains the oldest of Web directories. 211 Yahoo also 
contains a search engine index. Around 2001 Google 
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was created by a company that no longer exists. 21 The 
success of Google has been unparallel. Google now 
ranks as the most popular search engine. 20,22 

Google’s success is based on the fact that its search 
engines are not only comprehensive but designed for 
maximum speed. Google’s search engines are critical 
for the survival of many internet-based businesses. 
Google results are not only used by Yahoo, but also 
many other web-based components such as Netscape. 
FAST Search, also known as ‘All the Web’, is 
another popular search engine. It contains a large 
index and was the first search engine to achieve the 
200 million results mark. 20,23 The remaining popular 
search engines include Ask, AOL search, and MSN 
search. 

In September of 2006, another desktop search engine 
service surfaced, known as ChaCha. The principle 
behind the evolution of ChaCha is for consumers to 
have access to an infinite amount of answers via the 
internet, even when they do not have access to the 
internet. When consumers need instant answers about 
politics, breaking news, health, sports and so on, they 
can simply text message a question to ChaCha via 
their cell phone. The difference between ChaCha and 
the other search engines is that it is operated by 
human guides to provide answers. The live-guide 
then returns a concise answer within minutes which 
may include a reference to a web-site. Because of 
being human-driven, ChaCha claims to offer high- 
quality results geared by human intelligence. 24 " 26 

In January 2008, ChaCha services advanced by 
becoming the first ‘human guided’ mobile text 
messaging service free to consumers. The concept is 
of great ingenuity: ask a question via text message, 
and receive a simple yet accurate answer. The goal is 
to provide an accurate answer within minutes via cell 
phone text message in less than 160 characters. 24,26 
The company advertises service by encouraging 
consumers to “ask a question from their mobile 
phone as if they’re talking to a smart friend.” 24 
ChaCha prides itself on claiming to be the ‘smart 
search engine’. Accuracy of answers from ChaCha 
have been reported between 93-95%, 26-27 however, 
the method of how ChaCha measures accuracy is not 
available. 

Accuracy of health education information 
disseminated by ChaCha 

According to ChaCha. com, there are over 55,000 
ChaCha guides, with thousands online at any given 
time, to answer the 9 million plus text questions they 
receive every month. 24 According to ChaCha 


Incorporation, ChaCha guides are intemet-savvy 
individuals who are equipped with Internet search 
training and tools to help answer questions. 28 Prior to 
collecting data, a ChaCha guide was consulted to 
determine how guides were prepared or assessed to 
be qualified. The goal was to obtain background 
information on ChaCha guides, assess qualifications 
and training of guides, ascertain how guides 
determine what information to send text users, and 
what, if any, consequences there are for providing 
incorrect information. 

A ChaCha guide applicant must complete a two-hour 
evaluation assessing how quickly and efficiently one 
can search the Internet for answers to questions, is 
assessed to see if they can succinctly put the answer 
in a 160 character text message. The only training 
guides received is on how to summarize complex 
answers into 160 character text messages. ChaCha 
recommends reputable websites for the guides to use 
and suggests avoidance of questionable websites. 
However, guides are welcome to use websites they 
feel most comfortable with. There is no penalty for 
providing misinformation to users. 

Thousands of guides may be on-line at any given 
time. If a guide would like to work, they just log in to 
the ChaCha Universe system to answer questions. 
Questions texted to ChaCha (242242) are routed by 
an expeditor to a guide. Guides may answer 
anywhere from 20 to 50 questions per hour 
depending on the time of day. ChaCha keeps a log of 
questions from each cell phone number for 2-3 days. 

Purpose of Study 

The purpose of this study was to assess the accuracy 
of health and wellness-related information provided 
by ChaCha , the mobile text messaging service. The 
results will identify whether or not the information 
disseminated by ChaCha should be used by the 
public in making health-related decisions. 

Methods 

To assess the level of accuracy and completeness of 
health information disseminated by the mobile text 
messaging service ChaCha, the study consisted of 
texting 40 pre-determined health-related questions to 
ChaCha, compiling answers, and analyzing answers 
and websites referred for accuracy and completeness. 
Using the nationally recognized 10 content areas of 
health, the team developed 40 questions. The 
questions were texted to ChaCha by the team over a 
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period of one week. Each question was asked twice 
verbatim, for a total of 80 text questions to ChaCha. 
The questions were asked at varying times of the day 
and staggered so that the same question was not 
asked at the same time. The 10 content areas of 
health can be found in Table 1. 

Accuracy was determined by comparing ChaCha 
responses to three of the most well-known and 
popular college health textbooks. 29 ’ 31 The textbooks 
were chosen because they are the three best-selling 
textbooks at the university level and are seen as 
appropriate and comprehensive in the field of health. 
The textbooks rely on evidence-based materials and 
provide an inclusive overview of personal health. 

The level of accuracy of responses was scored using 
a Likert-type rubric ranging from 1 ( did not answer 
question or completely inaccurate), 2 (answer mostly 
inaccurate and incomplete), 3 (answer partially 
accurate), 4 (answer mostly accurate yet left out key 
points), and 5 (answer accurate and complete). 
Responses were considered superior (5) only if the 
message provided was entirely accurate and did not 
lack any vital information. Websites referred by 
ChaCha were analyzed in the same manner. 
Inter-rater reliability analysis 

To establish inter-rater reliability, two team members 
reviewed and assessed all ChaCha responses. The 
remaining two team members reviewed and assessed 
a random sample of ChaCha responses. All four 
reviewers have a minimum of a Master’s degree in 
health and teach personal health at the university 
level. Each question was rated a minimum of three 
times. Comparison of overall assessment scores were 
examined between reviewers. Team members rated 
ChaCha responses identically or within one Likert 
point 92.7% (78/80) of the time indicating a high 
level of internal consistency. 

This study received full Institutional Review Board 
(IRB) approval and followed all IRB protocol. 


Results 

To determine the extent to which ChaCha 
disseminated accurate and complete answers to 
health related questions, a rubric ranging from 1 
(inaccurate and incomplete) to 5 (accurate and 
complete) were used as the evaluation tool. Table 2 
reveals the frequencies and percentages of the 
ratings. Twenty-two and a half percent of the answers 
were given a rating of 1, 12.5% of the answers were 


given a rating of 2, 21.25% of the answers were 
given a rating of 3, 31.25% were given a rating of 4, 
and 12.5% of the answers were given a rating of 5. 
According to the results of the inter-rater reliability, 
there were 6 questions that received inconsistent 
ratings by all researchers, or 7.5 % (refer to Table 2). 
Ten out of the 80 questions asked were rated a 5 
(accurate and complete) while 18 out of the 80 
questions were given a 1 (inaccurate and incomplete). 

The ratings were also assessed within each health 
content area: emotional and mental health; family 
living; growth and development; nutrition; personal 
health and physical activity; alcohol, tobacco, and 
other drugs; communicable and chronic diseases; 
injury prevention and safety; consumer health; and 
environmental health. Notably, the areas of family 
living, growth and development, and nutrition 
received no ratings of 5 by the raters while consumer 
health and environmental health received neither a 
rating of 4 nor a rating of 5. In contrast, the area of 
injury prevention and safety received no rating of 1 
while the area of communicable and chronic diseases 
received neither a rating of 1 nor a rating of 2 (see 
Table 2). These results showed ChaCha 
disseminating the most accurate information when 
answering questions regarding communicable and 
chronic diseases and the least accurate information 
when answering questions regarding consumer health 
and environmental health. 

Half of the content areas contained at least one rating 
of 5 (complete and accurate). An example from the 
mental and emotional health content area is: How 
common are mental disorders? The response 
provided by ChaCha was, “An estimated 26.2% of 
adult Americans ages 18 and older suffer from a 
diagnosable mental disorder in a given year.” This 
response was rated a 5 (complete and accurate) 
because it was consistent with the information 
provided by all three textbooks. 29 ' 31 Seven of the 
content areas contained at least one rating of 1 
(incomplete and inaccurate). An example from the 
alcohol, tobacco, and other drugs content area is: Is 
there any cure for a hangover? The response provided 
by ChaCha was, “Consume foods and drinks that 
contain fructose (such as fruit juice or honey). This 
will help your body burn the alcohol.” This response 
is not accurate or complete. According to two of the 
textbooks, there is no “cure” for a hangover. In order 
to limit the effects of a hangover, the only option is to 
not drink alcohol or to not drink in excess. 30 " 31 See 
Table 3 for sample questions and text responses 
rating 1-5. 
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When ChaCha referred researchers to a website, the 
website was assessed for accuracy and completeness 
of the information. The same ratings used to assess 
text responses were used to assess the websites. Only 
five of the 80 text responses had a website referral. 
The ratings given to the websites were, 5, 4, 4, 4, and 
2. This showed that while the websites did provide 
slightly more accurate and complete information, 
they were not websites that could be used by 
consumers to get the best possible answer to their 
health-related questions. 

Of the five websites provided by ChaCha, two were 
in the content area family living. The website 
http://www.webmd.com/sex/birth-control/forgot-to- 
take-your-birth-control-pills was provided in 
response to the question, “What should I do if 1 
forgot to take a birth control pill for a day?” WebMd 
offered complete and accurate information, and was 
given a rating of 5, for the question advising the 
consumer to take the pill as soon as she remembers. 
Also, the website urges the consumer to call a 
healthcare provider, read the instructions on the birth 
control pack, and always remember to use another 
form of contraception until the pill pack is finished. 
This information is consistent with the answer 
provided by the textbooks. 29 ’ 31 

Another website, http://www.hairanalysistest.com/, 
was provided as a response to the question, “Should I 
get my hair analyzed to find out if I have nutrient 
deficiencies?” This question was included in the 
nutrition content area. The website was rated 2 
(mostly inaccurate and incomplete) by the 
researchers because it was giving inaccurate 
information, possibly to promote a product. The last 
two websites were in response to questions from the 
environmental health content area. 

Conclusions 

Data analyses indicated ChaCha is not a credible 
source for health-related information. ChaCha ’s 
responses were cross-referenced with three of the 
most widely recognized and used health textbooks in 
the college setting, and although these textbooks may 
not be complete and accurate in all aspects, they were 
chosen as the best framework to be used for this 
study. The ChaCha responses results yielded low 
scores for both accuracy and completeness. With 
certainty, we can conclude ChaCha ’s ability to 
respond accurately to specific health related 
questions is dubious and is not consistent with 
evidence -based research. 


Results indicated only 12.5% were rated accurate and 
complete, while 22.5 % of ChaCha 's responses were 
rated completely inaccurate/incomplete. This finding 
is contradictory to ChaCha’ s self-reported 93% 
accuracy rate. 26 ’ 27 Given the maximum 160 character 
text message allowance, ChaCha had either not given 
enough information to accurately answer the question 
or had given incorrect information that they received 
from an unreliable source. 

ChaCha was most accurate when disseminating 
information regarding communicable diseases. The 
answers are not noteworthy, but the results do show 
minor strength to the knowledge one can receive 
within a text message. Conversely, ChaCha was least 
accurate when disseminating information regarding 
chronic diseases. The inconsistent answers to the 10 
health content areas suggest ChaCha is not a reliable 
source to answer important health and wellness 
questions. 

The second part of the study investigated the extent 
to which the websites ChaCha refers are accurate and 
complete. Only 6% of the responses came with a 
website referral. The website ratings yielded a higher, 
or more accurate, result than text alone, however it 
was not remarkable. This suggests consumers will 
obtain a more detailed answer through the website 
referrals, unfortunately there is no guarantee the 
website is accurate and complete. Our findings are 
congruent with various studies ’ " ’ that question 
the accuracy and quality of health information on the 
Internet. 

In summary, while ChaCha provides a fast, efficient 
method to find answers, the accuracy of the answers 
is limited at best. The vast majority of the answers to 
our health-related questions were only partially 
accurate, with key points omitted. Omitting key 
points, such as using alternative methods of birth 
control after missing a birth control pill, could have 
potentially devastating results. In addition, 
misinformation may encourage consumers to spend 
money on something completely unfounded, such as 
a hair analysis for nutrient deficiencies. If the 
difference between being health literate and health 
illiterate means being able to process, understand, 
and make appropriate decisions with health 
information, then it could mean the difference 
between taking the recommended dosage of a 
medication and taking a lethal dosage. 32 This 
information, combined with the fact that a large 
proportion of the American population is lacking any 
type of health literacy training, leads the researchers 
to be concerned about the impact of Americans 
utilizing technologies such as ChaCha. How will the 
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American public determine what is considered valued 
data when they do not have any health literacy 
training? While ChaCha provides a fast, efficient 
method to find answers, the accuracy and 
completeness of the answers is questionable and not 
consistent with evidence-based research. The 
findings are consistent with previous research 
questioning the accuracy and quality of health 
information on the Internet. Translation to Health 
Education Practice: Health educators can use the 
results of this study to alert the public that ChaCha is 
not a reliable source for health and wellness-related 
information. 
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Table 1. 10 Content Areas of Health 


Mental & Emotional Health 
Family Living 
Growth & Development 
Nutrition 

Personal Health & Physical Activity 
Alcohol, Tobacco & Other Drugs 
Communicable & Chronic Diseases 
Injury Prevention & Safety 
Consumer Health 
Environmental Health 
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Table 2. Range of Accuracy (Percentage) Among Ten Content Areas 


Content Area 
and Average 
Score 

Inaccurate 

Mostly 

Inaccurate 

Partially 

Accurate 

Mostly 

Accurate 

Accurate 

Overall 

22.5 

12.5 

23.8 

21.2 

12.5 

Mental & 
Emotional 

50.0 

0.0 

0.0 

25.0 

25.0 

Family Living 

25.0 

25.0 

0.0 

50.0 

0.0 

Growth & 
Development 

12.5 

25.0 

25.0 

37.5 

0.0 

Nutrition 

50.0 

0.0 

25.0 

25.0 

0.0 

Personal Health 
& Physical 
Activity 

25.0 

12.5 

0.0 

37.5 

25.0 

Alcohol, 

Tobacco, & Other 
Drugs 

50.0 

0.0 

25.0 

12.5 

12.5 

Communicable & 
Chronic Diseases 

0.0 

0.0 

57.14 

14.29 

28.57 

Injury Prevention 
& Safety 

0.0 

25.0 

25.0 

12.5 

37.5 

Consumer Health 

16.67 

50.0 

33.33 

0.0 

0.0 

Environmental 

Health 

0.0 

0.0 

100.0 

0.0 

0.0 
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Table 3. Sample Questions, Responses, and Ratings 


Rating 

Question Texted 

ChaCha ’s Response 

Explanation of Rating 

1 

Is there any cure for a 
hangover? 

“Consume foods and drinks 
that contain fructose (such as 
fruit juice or honey). This will 
help your body burn the 
alcohol.” 

Inaccurate and incomplete. According 
to two of the textbooks, there is no 
“cure” for a hangover. In order limit 
the effects of a hangover, the only 
option is to not drink alcohol or to not 
drink in excess. 

2 

What is an HMO? 

“An HMO, Health 
Maintenance Organization, is 
a specific type of health care 
plan found in the United 
States.” 

Mostly inaccurate and doesn’t give 
any specific information. According 
to the textbooks, an HMO is a type of 
managed care plan that provides a 
wide range of covered health benefits 
for a fixed amount prepaid by the 
consumer. 

3 

What is 

photochemical smog, 
and should I be 
worried about it? 

“Photochemical smog is a 
mixture of pollutants which 
includes particulates, nitrogen 
oxides, ozone, aldehydes, 
peroxethanoyl.” 

Partially accurate. This response, 
however, only answers the first half of 
the question. 

4 

What should I do if I 
forget to take a birth 
control pill for a day? 

“When you forget, take it as 
soon as you remember. If you 
don't remember until the next 
day, go ahead and take 2 pills 
that day.” 

Mostly accurate. This answer is 
accurate, but it leaves out one piece of 
important information. According to 
the text books, it is necessary to use 
another form of contraception in order 
to prevent pregnancy. 

5 

How common are 
mental disorders? 

“An estimated 26.2% of adult 
Americans ages 18 and older 
suffer from a diagnosable 
mental disorder in a given 
year.” 

Complete and accurate. This answer 
is consistent with the information 
provided by all three text books. 
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